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Polarization  Vision 


1.1  Introduction 

One  of  the  fundamental  properties  of  light  is  polarization.  Polarization  of  light  radiation 
has  been  used  as  an  important  measurement  tool  for  the  study  of  a  number  of  phenomena 
in  astronomy,  optics,  and,  crystallography.  Only  recently  has  the  measurement  of  polar¬ 
ization  been  considered  for  use  in  imagery  [1],  [2],  [3].  In  [4]  a  number  of  research  results 
were  presented  demonstrating  that  the  sensing  of  reflected  polarization  can  directly  enhance 
the  capabilities  and/or  simplify  certain  important  Image  Understanding  tasks  beyond  what 
intensity  and  color  can  accomplish  alone.  This  was  shown  to  be  true  for  tasks  such  as  di¬ 
electric/metal  discrimination  in  images,  detection  of  specularities  (i.e.,  glossy  highlights), 
quantitative  separation  of  specular  and  diffuse  reflections,  and,  direct  identification  of  oc¬ 
cluding  contours  (i.e.,  edge  boundaries  between  object  and  background).  Furthermore  while 
these  were  a  diverse  set  of  applications,  all  these  methods  arose  from  the  same  physical  prin¬ 
ciples  governing  the  polarization  of  reflected  light  from  both  dielectric  and  metal  surfaces. 
The  key  new  perspective  that  came  out  of  this  work  with  respect  to  the  role  of  polarization 
sensing  in  Image  Understanding  was  that  polarization  being  orthogonal  to  wavelength  (i.e., 
color)  and  a  more  general  physical  characteristic  of  light  than  intensity  can  carry  additional 
information  and  therefore  provide  a  richer  description  of  an  imaged  scene.  Therefore,  the 
use  of  polarization  for  Image  Understanding  is  an  augmentation  of  sensed  light  parameters 
from  a  scene  that  can  have  significantly  more  capabilities  than  intensity  and  color  alone. 
“Polarization  Vision”  is  an  area  that  could  potentially  play  a  mainstream  role  in  Computer 
Vision. 

The  practical  significance  of  the  enhanced  capabilities  of  Polarization  Vision  was  still 
hindered  by  the  fact  that  commercially  available  video  imaging  cameras  are  geared  to  sense 
only  intensity  and  color.  The  sensing  of  polarization  requires  optical  components  that  are  not 
available  in  existing  video  cameras  for  Computer  Vision.  Past  results  have  almost  always 
used  a  monochrome  or  color  CCD  camera  with  a  mechanically  rotating  linear  polarizing 
filter  placed  in  front  of  the  camera  lens.  Multiple  images  have  been  obtained  with  respect  to 
different  orientations  of  the  polarizing  filter-  at  least  three  polarization  component  images 
are  required  for  a  complete  measurement  of  partial  linear  polarization.  While  producing 
some  good  results,  this  design  for  a  polarization  imaging  device  is  cumbersome  both  from  a 
practical  and  a  conceptual  standpoint.  The  mechanical  rotation  of  an  optical  filter  in  front 
of  an  imaging  camera  causes  slight  geometric  shifting  of  the  projection  of  the  world  scene 
onto  the  focal  plane,  and  where  there  are  significant  local  intensity  variations  in  a  scene  (e.g., 
edges)  the  shifting  of  these  intensity  variations  across  pixels  between  polarization  component 
images  causes  significant  error  in  polarization  measurement.  At  least  as  important,  real-time 
applications  outside  the  laboratory  are  difficult  using  this  type  of  design.  The  mechanical 
rotation  of  a  polarizing  filter  gives  Polarization  Vision  the  appearance  of  being  a  multiple 
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view  modality,  when  in  fact  conceptually  the  measurement  of  polarization  need  not  involve 
multiple  views  any  more  than  the  measurement  of  color. 

In  [4]  a  high-level  description  was  given  for  a  Polarization  Camera  that  would  fully  au¬ 
tomatically  sense  polarization  components  of  light  in  a  passive  fashion  without  mechanical 
manipulation  of  external  optics.  This  would  make  the  enhanced  capabilities  of  Polariza¬ 
tion  Vision  accessible  to  a  larger  diversity  of  Image  Understanding  applications.  The  first 
working  prototype  of  a  Polarization  Camera  was  presented  in  [5],  [6],  [7]  utilizing  liquid  crys¬ 
tal  technology.  This  was  presented  in  conjunction  with  a  visually  intuitive  representation 
scheme  for  partial  linear  polarization  images.  In  the  1970s  biologists  Bernard  and  Wehner  [8] 
conjectured  that  polarization  vision  in  certain  animals  could  be  analogous  to  color  vision  in 
humans.  In  [5],  [6]  a  representation  scheme  was  developed  from  this  conjecture  for  measured 
partial  linear  polarization  at  pixels  in  digital  images  in  terms  of  the  saturation  of  a  color 
hue.  To  Computer  Vision  researchers  this  was  a  significant  step  because  instead  of  measured 
polarization  parameters  being  just  a  set  of  abstract  numbers,  humans  can  actually  “see”  Po¬ 
larization  Vision  measured  by  a  polarization  camera  as  being  a  natural  visual  domain  unto 
itself.  An  overview  of  a  number  of  prototype  designs  for  Polarization  Cameras  are  presented 
in  [9], 

This  subsection  overviews  significant  new  developments  in  Polarization  Vision  resulting 
from  the  application  of  Polarization  Camera  Technology  to  a  number  of  real  practical  prob¬ 
lems  for  Image  Understanding  outdoors  and  underwater.  The  system  advancements  that 
have  made  these  applications  possible  are  the  low-cost  modular  designs  of  compact  opti¬ 
cal  and  electronic  components  that  can  be  used  with  almost  any  kind  of  imaging  device. 
Utilizing  our  visualization  scheme  for  polarization  images,  Polarization  Cameras  are  giving 
humans  for  the  first  time  a  comprehensive  sensory  view  of  a  whole  new  visual  domain  which 
is  speculated  to  be  as  broad  an  area  in  both  theoretical  and  practical  significance  as  color 
vision  is  an  extension  to  intensity  vision. 


1.2  Brief  Polarization  Background 

The  left  of  Figure  1  shows  the  electric  field  distribution  of  partially  linear  polarized  light 
viewed  head-on  as  the  superposition  of  an  unpolarized  component  consisting  of  an  isotropic 
electric  field  distribution,  with  a  linear  polarized  component  consisting  of  an  electric  field  ori¬ 
ented  along  a  single  axis.  Humans  only  visually  perceive  the  energy  flux  of  this  distribution 
as  ’’intensity”.  What  is  not  visually  perceived  are  the  relative  magnitudes  of  unpolarized 
and  linear  polarized  components  that  constitute  the  light,  nor  the  orientation  of  the  axis  of 
linear  polarization  if  there  is  such  a  non-zero  component.  The  relative  proportion  of  linear 
polarization  we  will  term  the  partial  polarization  varying  between  0  (completely  unpolarized 
light)  and  1  (completely  linearly  polarized  light).  Partial  polarization  is  also  known  as  the 
degree  of  polarization  [10].  The  orientation  of  linear  polarization  we  will  term  the  phase  or 
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Figure  1:  Hue-saturation-intensity  representation  scheme  for  partial  linear  polarization. 
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Figure  2:  Rotation  of  the  linear  polarized  component  of  polarization  by  a  TN  liquid  crystal. 
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Figure  3:  Schematic  of  an  optical  head  and  electronic  box  interfaced  with  an  NTSC  Imaging 
Device  giving  it  automatic  polarization  sensing  capability. 
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simply  the  orientation  of  polarization.  Note  that  the  phase  varies  within  the  range  0  —  180°. 
Together  the  parameters  (i)  intensity,  (ii)  partial  polarization,  and,  (iii)  phase,  completely 
determine  the  state  of  partial  linear  polarization.  Parameters  (ii)  and  (iii)  are  the  sensory 
augmentation  beyond  conventional  intensity  and  color  vision.  A  precise  definition  of  polar¬ 
ization  is  presented  in  [10]. 

In  general  polarization  of  light  can  have  a  fourth  parameter  which  describes  an  ellip- 
tically  polarized  light  component  of  which  linear  polarization  is  a  particular  case.  There 
are  some  benefits  to  measuring  this  fourth  parameter  of  polarization,  as  explained  in  [1]  for 
determination  of  surface  orientation,  and  in  [4]  for  dielectric/metal  discrimination.  However, 
studies  by  Waterman  [11]  and  other  researchers  have  shown  that  almost  all  naturally  occur¬ 
ring  light  outdoors  and  underwater,  scattered  and  reflected,  as  well  as  light  in  most  indoor 
environments  is  partially  linear  polarized.  While  we  have  extended  the  designs  for  Polar¬ 
ization  Cameras  to  measure  all  four  polarization  parameters,  we  have  found  little  reason  in 
our  applications  so  far  to  use  such  designs.  Polarization  Cameras  that  measure  partial  linear 
polarization  have  proven  to  be  quite  sufficient. 

A  state  of  partial  linear  polarization  can  be  measured  using  a  linear  polarizing  filter  which 
resolves  the  electric  field  distribution  of  light  along  a  given  orientation.  At  a  pixel  in  an  image 
the  transmitted  radiance  of  partially  linear  polarized  light  through  a  linear  polarizing  filter 
varies  sinusoidally  with  period  180°  as  a  function  of  polarizer  orientation.  The  maximum, 
Imaxi  of  the  sinusoid  occurs  when  polarizer  orientation  is  aligned  parallel  to  the  orientation 
of  the  linear  polarized  component,  and  the  minimum,  Imin,  occurs  when  polarizer  orienta¬ 
tion  is  aligned  perpendicular  to  the  linear  polarized  component.  With  no  linear  polarized 
component  the  sinusoid  is  simply  a  horizontal  line  with  one-half  the  intensity  magnitude  of 
the  unpolarized  light.  With  no  unpolarized  component  the  minimum  value  of  the  sinusoid  is 
0.  Since  a  sinusoid  can  be  uniquely  determined  by  three  points,  three  transmitted  radiance 
measurements  can  be  taken  between  0°  and  180°  to  determine  partial  linear  polarization.  We 
have  standardly  taken  measurements  at  0°,  45°,  and  90°.  Representing  these  measurements 
respectively  by  Iq,  h^,  and,  /go,  the  three  parameters  of  partial  linear  polarization  can  be 
derived  according  to: 


phased  =  (1/2)  tan  ^  (  °  — - — —)  -I-  90,  (1) 

ha  —  h 

if  (/go  <  /o)  [  if  (/45  <  /o)  ^  0  +  90  else  0  =  0  -  90] 

intensity  =  /o-f-/go,  (2) 

partial  polarization  =  .  (3) 

In  previous  publications  the  convention  has  been  used  that  the  phase  angle  6  represent 
the  angle  at  which  the  minimum  Imin  is  observed  for  the  sinusoid  of  transmitted  radiance. 
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relative  to  the  reference  0°.  Geometrically  this  makes  the  phase  angle  perpendicular  to  the 
orientation  of  the  linear  polarized  component.  In  this  paper  we  add  a  90°  term  in  expression 
(1)  using  the  more  intuitive  convention  that  makes  the  phase  angle  parallel  with  the  linear 
polarized  component. 

Figure  1  shows  a  mapping  of  partial  linear  polarization  into  a  hue-saturation-intensity 
visualization  scheme  proposed  in  [5],  [6].  Phase  and  partial  polarization  which  are  not 
observed  by  human  vision  are  represented  respectively  in  terms  of  hue  and  saturation.  Since 
phase  of  polarization  is  in  the  range  0-180°,  phase  angle  is  multiplied  by  two  to  map  into 
the  entire  range  of  hue  representation.  An  image  for  which  the  measurement  of  polarization 
at  each  pixel  is  represented  according  to  the  scheme  depicted  in  Figure  1  will  be  termed  a 
partial  linear  polarization  image,  or  simply  a  polarization  image.  Chromaticity  at  a  pixel  in 
a  polarization  image  means  that  there  is  some  presence  of  linear  polarization  as  saturation 
corresponds  directly  to  partial  polarization.  Unpolarized  light  at  a  pixel  in  a  polarization 
image  is  therefore  achromatic.  Intensity  is  of  course  simply  intensity  itself. 

We  analyzed  monochrome  polarization  images,  meaning  that  the  measured  polarization 
state  at  each  pixel  is  taken  over  a  single  range  of  wavelengths.  Of  course  multiple  polariza¬ 
tion  images  can  be  respectively  taken  over  different  spectral  wavelength  ranges  adding  even 
more  sensed  information.  It  can  be  difficult  to  visualize  a  vision  system  that  is  simultane¬ 
ously  sensitive  to  intensity,  color,  and,  polarization.  Photometrically  human  vision  is  three 
dimensional  at  each  spatial  point-  two  dimensions  for  color  (i.e.,  hue  and  saturation),  and 
one  dimension  for  brightness.  Adding  partial  linear  polarization  sensing  would  make  for  a 
vision  system  which  is  five  dimensional.  It  turns  out  that  some  animals,  such  as  the  Mantis 
Shrimp  [12],  may  possess  such  higher  dimensional  photometric  vision  systems.  Images  simul¬ 
taneously  visualizing  intensity,  color  and  polarization  can,  for  instance,  incorporate  texture 
or  dithering  patterns  at  pixels  that  represent  states  of  polarization. 

There  are  two  physical  principles  of  reflected  polarization  which  are  basic  to  Polarization 
Vision  [4],  [13].  The  plane  of  incidence  at  an  object  point  viewed  by  a  camera  sensor  is 
determined  by  the  surface  normal  at  the  point  and  the  viewing  direction. 

•  For  unpolarized  incident  light,  specular  reflection  (i.e.,  glossy  reflection)  has  non-zero 
partial  polarization,  except  for  normal  and  grazing  incidence,  with  linear  polarized 
component  oriented  perpendicular  to  the  plane  of  incidence. 

•  Diffuse  reflection  from  dielectric  surfaces  for  most  viewing  angles  is  essentially  unpo¬ 
larized,  becoming  significantly  partially  polarized  at  greater  than  60°  with  respect  to 
the  angle  between  the  viewing  direction  and  the  surface  normal.  The  orientation  of  the 
linear  polarized  component  is  parallel  to  the  plane  of  incidence  for  diffuse  reflection. 
For  smooth  dielectric  surfaces  this  is  true  for  any  incident  polarization  state. 
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1.3  Portable  Polarization  Camera  Sensors  From  Modular  Com¬ 
ponents 

In  [5],  [6]  a  prototype  Liquid  Crystal  Polarization  Camera  was  developed  for  full  automatic 
accurate  capture  of  polarization  imagery  in  the  laboratory  under  computer  control.  The 
electro-optical  switching  of  Twisted  Nematic  (TN)  liquid  crystals  in  front  of  a  fixed  linear 
polarizer  obviates  the  need  for  mechanically  rotating  the  linear  polarizer  for  obtaining  po¬ 
larization  component  images  with  an  intensity  video  camera.  When  an  AC  signal  is  passed 
through  each  of  the  liquid  crystals,  the  polarization  state  of  light  passing  through  the  crystals 
is  unaltered  -  with  no  voltage  the  linear  polarized  component  of  incident  light  is  rotated  by 
a  fixed  amount  n°,  the  value  n  determined  when  the  TN  liquid  crystal  is  fabricated.  See 
Figure  2.  The  two  TN  liquid  crystals  used  are  respectively  45°  and  90°  twist.  A  timing 
circuit  places  the  switching  of  the  liquid  crystals  in  synchronization  with  camera  video  rate, 
and  in  series  the  two  liquid  crystals  can  produce  0°,  45°,  90°  and  135°  twist  states  at  which 
times  an  image  is  digitized  -  only  the  first  three  states  are  used  to  completely  measure  partial 
linear  polarization.  W^hat  used  to  be  done  by  mechanical  rotation  of  a  linear  polarizing  filter 
is  now  done  electro-optically  using  liquid  crystals  and  a  fixed  linear  polarizing  filter,  which 
does  not  produce  optical  distortion.  The  switching  and  grabbing  of  polarization  component 
images  is  placed  under  full  electrical/computer  control.  The  switching  of  the  liquid  crystals 
in  synchronization  with  video  framerate  insures  that  polarization  components  are  imaged 
when  the  liquid  crystals  are  in  a  fully  “relaxed”  state  as  well  as  knowing  which  liquid  crystal 
state  corresponds  to  each  imaged  polarization  component.  The  original  TN  liquid  crystals 
in  [5],  [6],  required  1/10  second  (three  1/30  second  video  frametimes)  to  switch  states  and 
then  a  fourth  video  frametime  to  grab  the  image.  With  twelve  video  frametimes  to  grab  a 
complete  set  of  three  polarization  component  images,  polarization  images  were  sensed  at  2.5 
Hz. 

Current  TN  liquid  crystals  switch  in  less  than  a  video  frametime-  now  only  one  frame¬ 
time  is  used  to  switch  states  and  one  frametime  to  grab  the  polarization  component  image. 
At  two  video  frametimes  to  sense  each  polarization  component  image,  15  component  images 
can  be  sensed  in  a  second.  With  six  video  frametimes  to  grab  a  complete  set  of  three  polar¬ 
ization  component  images,  polarization  images  can  now  be  sensed  at  5  Hz  (one  polarization 
image  every  0.2  seconds).  If  only  fields  are  used  at  half  image  resolution  (the  interlacing  of  a 
field  consisting  of  even  numbered  raster  lines,  and  a  field  consisting  of  odd  numbered  raster 
lines,  makes  up  an  entire  frame)  the  speed  doubles  to  lOHz  for  the  sensing  of  partial  linear 
polarization  images.  The  design  for  the  original  prototype  Liquid  Crystal  Polarization  Cam¬ 
era  has  been  adapted  for  portable  use  outdoors  and  underwater  by  designing  modular  and 
compact  components  that  are  compatible  with  any  standard  NTSC  video  imaging  device. 
The  two  main  modular  components  that  have  been  designed  are; 

•  (I)  the  coupling  of  the  liquid  crystals  and  fixed  linear  polarizer  to  form  a  small  optical 
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head  that  can  be  easily  mounted  on  any  camera  lens, 

•  (II)  the  design  of  a  compact  electronic  box  that  drives  the  liquid  crystals  in  synchro¬ 
nization  with  the  video  rate  of  the  NTSC  imaging  device,  as  well  as  corresponding 
polarization  component  images  with  the  state  of  the  liquid  crystals  when  they  were 
imaged. 

TN  liquid  crystals  are  low-power  only  requiring  a  9-volt  battery  to  operate  the  electronics 
that  drive  them.  Furthermore  these  liquid  crystals  are  relatively  low  cost.  Figure  3  shows  a 
high  level  schematic  of  how  the  optical  head  and  electronic  box  are  interfaced  with  an  NTSC 
video  imaging  device. 

A  number  of  processing  options  utilizing  equations  (1),  (2),  and,  (3),  have  been  used 
depending  upon  suitability  for  particular  applications.  When  using  a  digitizing  board  along 
the  backplane  of  a  host  computer  two  possibilities  are  useful;  (i)  a  single  sequence  of  three 
polarization  component  images  are  sensed,  digitized  and  then  processed  to  produce  a  single 
polarization  image;  (ii)  a  sequence  of  polarization  images  can  be  computed  on  the  fly  using 
real-time  processor  hardware  (e.g,  Datacube  MV-200  can  incorporate  equations  (1),  (2), 
and,  (3)  into  look-up  tables)  with  either  a  polarization  image  updated  5  times  a  second 
after  the  completion  of  each  sequence  of  0°,  45°,  90°  polarization  component  images,  or,  a 
piplineing  scheme  in  which  a  polarization  image  is  updated  15  times  a  second  processing  the 
most  recently  sensed  three  polarization  component  images  each  time.  The  15Hz  polarization 
image  capability  of  this  pipelining  scheme  is  deceptive  though-  each  element  in  a  scene 
must  remain  relatively  static  for  1/5  second  (6  video  frametimes)  for  a  complete  sequence  of 
three  polarization  images  to  be  sensed,  otherwise  polarization  measurement  will  be  in  error. 
Partial  linear  polarization  sensing  is  still  at  5Hz  with  respect  to  a  scene  in  which  objects  are  in 
movement.  Using  a  video  camcorder  polarization  component  images  are  stored  on  video  tape 
which  can  then  be  later  digitized  and  processed  back  in  the  laboratory  using  schemes  (i),  or, 
(ii)  discussed  above.  Since  camcorders  are  very  portable,  and  a  large  amount  of  image  data 
can  be  stored  on  a  single  video  tape,  they  provide  a  very  convenient  polarization  camera 
for  held  use.  The  only  drawbacks  to  using  a  video  camcorder  is  slightly  less  polarization 
image  quality  due  to  the  double  process  of  sensing  off  of  video  tape  and  then  digitizing,  and, 
Hi-8  high  resolution  video  format  is  limited  to  400  scanlines.  Nonetheless  we  have  obtained 
excellent  results  using  camcorders. 

Figure  4  shows  a  portable  version  of  a  Liquid  Crystal  Polarization  Camera  using  a  small 
HI-8  Video  Camcorder.  The  two  modular  components  described  above  can  be  clearly  seen 
here  with  the  liquid  crystal/polarizer  optical  head  mounted  on  the  Camcorder,  and  the 
compact  white  electronic  box  containing  the  timing  circuitry  that  synchronizes  and  drives 
the  liquid  crystals,  to  the  right. 

Figure  5  shows  a  similar  liquid  crystal/linear  polarizer  optical  head  mounted  on  a  CCD 
video  camera  that  has  been  encased  in  a  waterproof  housing.  This  was  developed  under  the 
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Figure  4:  Liquid  Crystal  Polarization  Camera 
using  a  Video  Camcorder. 


Figure  5:  Liquid  Cr>'stal  Polarization  Camera 
for  undenvater  use.  sponsored  by  U.S.  Coast 


Figure  12:  Intensity  image  of  grass  and  dirt 
terrain  with  protruding  rock. 


Figure  14:  Intensity  image  of  camouflaged 
HMMWV  partially  occluded  by  a  bush. 


Figure  16:  Intensity  image  of  a  HMMWV. 


Figure  13:  Polarization-based  segmentation 
of  dirt  (Red)  and  protruding  rock  (Green). 


Figure  15:  Polarization-based  segmentation 
of  highly  polarization  reflecting  camouflage 
resulting  in  target  detection  (Magenta). 


Figure  17:  Polarization  image  of  HMMWV. 
Polarization  features  provide  additional 
target  recognition  capability. 


Figure  20:  Intesnity  image  of  rusted  and 
scrape  steel  panel. 


Figure  21:Polari7.ation  based  segmentation  of 
rust  (Yellow)  and  superficial  scrapes  (Magenta). 


Figure  24:  Polarization  image  of  Mantis  Shrimp  underwater. 
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sponsorship  of  the  U.S.  Coast  Guard,  and  is  being  used  at  the  Naval  Surface  Warfare  Center, 
Annapolis,  Maryland,  in  underwater  test  tanks  and  river  basins  to  study  automated  ship 
hull  damage  assessment.  In  this  setup  a  sequence  of  three  polarization  component  images 
are  digitized  and  stored  on  a  host  computer,  which  are  then  later  processed. 

1.4  Application  Areas 

The  development  of  portable  Liquid  Crystal  Polarization  Cameras  has  facilitated  the  study 
of  a  number  of  new  application  areas  in  Polarization  Vision  revealing  potential  practical 
utility  for  Image  Understanding,  and,  as  a  potentially  valuable  tool  for  the  general  study  of 
science.  On-going  research  is  shown  in  four  of  these  application  areas.  These  are  just  a  small 
subset  of  a  number  of  compiled  results  in  these  areas,  and  are  not  meant  to  be  completely 
conclusive-  an  entire  paper  can  be  easily  devoted  to  each  application.  The  results  presented 
are  meant  to  reveal  some  of  the  salient  capabilities  of  Polarization  Vision  for  real  applications 
which  is  motivating  further  study  and  the  potential  utility  of  Polarization  Vision  as  a  new 
promising  approach  to  Image  Understanding.  The  fundamental  physics-based  principles 
upon  which  the  application  results  are  based  are  summarized.  Color  in  some  of  the  following 
figures  are  used  to  represent  polarization  information,  while  in  other  figures  color  is  used  as 
a  segmentation  label.  Figures  6  thru  13  were  sensed  using  the  camera  in  Figure  4,  digitized 
and  then  post  processed.  Figures  14  thru  17  were  sensed  using  a  liquid  crystal  polarization 
camera  mounted  on  top  of  a  HMMWV  at  Martin-Marietta  Denver.  Figures  18  thru  21 
were  sensed  using  the  camera  shown  in  Figure  5.  Figures  22  thru  24  were  sensed  using  a 
waterproofed  version  of  the  camera  in  Figure  4. 

1.4.1  Polarization  in  Nature  and  Natural  Object  Recognition 

Figures  6  and  7  respectively  show  an  intensity  image,  and,  a  hue-saturation-intensity  repre¬ 
sentation  of  the  partial  linear  polarization  image  for  a  beach  scene  at  sunrise,  the  sun  is  rising 
far  to  the  left  out  of  the  field  of  view.  The  partial  linear  polarization  image  reveals  a  number 
of  physical  phenomena  at  work  in  nature  that  intensity  and  color  vision  are  oblivious  too. 
The  vast  majority  of  significant  partial  polarization  in  this  scene  comes  from  either  ocean 
or  sky.  The  laws  of  Rayleigh  scattering  of  sunlight  off  of  atmosphere  dictate  that  skylight 
is  nearly  unpolarized  in  the  vicinity  of  the  sun  becoming  more  and  more  partially  polarized 
moving  angularly  away  from  the  sun.  According  to  Rayleigh  the  partial  polarization  from  a 
patch  of  skylight  at  angle  a  from  the  sun  is  given  by  [10]: 

sin^  a 
1  +  cos^  Q 

the  orientation  of  partial  linear  polarization  being  tangent  to  the  circle  centered  at  the  sun 
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and  passing  through  the  patch  of  sky.  There  is  some  depolarization  caused  by  water  and 
haze  particles  as  discussed  by  Waterman  [11].  In  Figures  6  and  7  the  skylight  in  this  field  of 
view  has  a  in  the  range  45°  —  55°  and  is  therefore  significantly  polarized.  This  scene  shows 
how  different  cloud  densities  effect  skylight  polarization,  and  what  happens  when  partially 
polarized  skylight  reflects  off  of  water.  Due  to  scattering  of  light  in  air  the  linear  polarization 
content  of  skylight  is  oriented  almost  vertically  in  the  image  shown  as  variations  in  Cyan  and 
Green  color  hue.  The  whiteness  of  the  denser  clouds  shows  that  they  are  unpolarized,  less 
dense  clouds  have  some  chromaticity  because  they  are  translucent  with  respect  to  background 
polarized  skylight  depolarizing  this  to  some  extent  but  also  slightly  altering  the  orientation 
of  partial  linear  polarization  of  skylight.  Some  portions  of  ocean  reflect  light  from  clouds, 
other  portions  of  ocean  reflect  light  from  sky.  Unpolarized  light  from  dense  clouds  reflects  off 
of  water  with  linear  polarization  component  oriented  horizontally  (Magenta)  while  almost 
vertical  orientation  of  linear  po’  j,i  ization  contained  in  skylight  reflects  off  of  water  within  the 
approximate  orientation  range  40  -  60°,  shown  in  Blue-Purple  color  hues.  The  maximum 
partial  polarization  in  Figure  7  is  45%. 

Figures  8  and  9  respectively  show  an  intensity  image,  and,  a  hue-saturation-intensity 
representation  of  the  partial  linear  polarization  image  of  another  beach  scene,  this  scene 
being  at  sunset.  The  sun  is  low  in  the  sky  just  out  of  view  slightly  to  the  left  of  the  image. 
The  range  of  a  in  this  view  is  from  10°  -  40°  and  the  qualitative  effect  of  increasing  partial 
polarization  as  a  function  of  increasing  ot  is  clearly  seen  moving  from  left  to  right  in  Figure 
9.  The  phase  of  partial  polarization  from  ocean  is  oriented  horizontal  (hue  Magenta)  in  the 
image  while  the  phase  of  partial  polarization  from  skylight  is  oriented  nearly  vertical  (hues 
Cyan  and  Green)  in  the  image.  The  horizontal  phase  of  reflected  polarization  from  ocean  is 
consistent  with  nearly  unpolarized  skylight  in  the  vicinity  of  the  sun  specularly  reflecting  off 
of  water  into  the  camera  (first  itemized  principle  at  the  end  of  Section  2).  The  maximum 
partial  polarization  in  Figure  9  is  just  under  40%. 

Quantitative  measurements  taken  with  our  polarization  camera  of  patches  of  “clear” 
sky  show  that  in  fact  partial  polarization  does  behave  according  to  the  Rayleigh  partial 
polarization  formula  to  within  10%-  partial  polarization  is  seen  to  be  just  a  little  less  than 
predicted  by  the  formula.  The  orientation  of  partial  linear  polarization  has  been  verified  to 
be  quite  accurate  to  within  ±5°.  Predictions  about  polarization  of  skylight  and  clouds  have 
so  far  been  primarily  theoretical,  and  Polarization  Cameras  are  providing  the  first  practical 
tool  with  which  to  empirically  study  measured  polarization  distributions  in  images  of  nature. 

Figure  10  shows  the  intensity  image  of  a  river  that  runs  between  wooded  hills  under 
a  cloudy  (unpolarized)  sky.  Water  is  difficult  to  segment  in  natural  scenes  using  intensity 
and  color  due  to  its  resemblance  to  other  elements  of  the  scene  from  direct  mirror  reflection. 
Figure  11  shows  the  corresponding  partial  linear  polarization  image  revealing  that  in  this  case 
practically  all  partial  polarization  comes  from  visible  portions  of  river  in  the  scene.  Strongest 
partial  polarization  in  Figure  11  is  30%.  The  phase  of  reflected  polarization  represented  by 
hues  of  Magenta-Red  depict  a  horizontal  phase  of  reflected  polarization  physically  consistent 
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with  specular  reflection  of  unpolarized  skylight  off  of  a  horizontal  surface  (first  itemized 
physical  principle  in  section  2).  Since  water  is  a  fluid  and  therefore  oriented  parallel  to  gravity 
this  provides  a  physically-based  methodology  for  segmenting  water  in  a  natural  scene. 

Figure  12  shows  grassy  terrain  with  intermixed  patches  of  dirt,  and  rocks  which  are  either 
strewn  about  or  are  protruding  out  from  dirt.  There  is  a  white  concrete  path  on  the  left  side, 
and  trees  in  the  background  underneath  a  cloudy  sky.  Exposed  dirt,  and  rock,  strongly  po¬ 
larize  light  upon  reflection  as  opposed  to  grass  which  typically  has  little  partial  polarization. 
Figure  13  shows  a  segmentation  of  dirt  and  possible  rock  using  partial  polarization  and  phase 
of  polarization  information.  Pixels  with  at  least  5%  partial  polarization  are  either  labeled 
Red  or  Bright  White.  (The  criterion  5%  represents  “significant”  partial  polarization  with 
a  high  confidence  that  the  measured  intensity  variations  between  polarization  components 
are  not  due  to  camera  noise.)  The  phase  of  polarization  for  Red  labeled  pixels  is  within  the 
range  of  ±20°  with  respect  to  horizontal,  representing  primarily  level  dirt  patches.  Bright 
White  labeled  pixels  in  Figure  13  have  phase  of  polarization  significantly  deviant  from  hori¬ 
zontal  (i.e.,  outside  the  range  ±20°)  implying  a  polarizing  surface  that  is  tilted  and  therefore 
most  probably  rock.  Because  of  the  first  itemized  physical  principle  described  at  the  end  of 
Section  2,  phase  of  polarization  from  specular  reflection  directly  reveals  the  orientation  of 
the  projected  surface  normal  onto  the  image  plane  (i.e.,  the  surface  normal  is  contained  in 
the  plane  of  incidence  for  smooth  objects).  Of  course  there  could  be  level  portions  of  rock 
within  the  Red  labeled  pixels,  but, rock  protrusions  must  eventually  have  a  significant  tilt  in 
some  parts.  The  Bright  White  pixels  in  Figure  13  do  truly  reveal  many  areas  where  there  is 
protruding  rock  or  small  freestanding  rocks,  including  a  significantly  sized  rock  protrusion 
near  the  lower  right  corner  of  this  image.  This  is  important  information  for  determining 
traversability  for  autonomous  land  vehicles. 

1.4.2  Automatic  Target  Detection  And  Recognition 

At  the  Lockheed-Martin,  Denver  facility  a  liquid  crystal  polarization  camera  has  been  in¬ 
stalled  atop  one  of  the  Semiautonomous  Surrogate  Vehicles  (SSV)  which  is  a  HMMWV 
capable  of  traversing  almost  any  kind  of  rugged  terrain.  The  goal  of  the  study  is  to  create  an 
empirical  polarization  image  database  of  various  Army  vehicles  in  various  poses  for  different 
viewer-solar  illumination  geometries.  From  the  augmented  polarization  feature  information 
contained  in  such  a  database,  the  study  is  examining  how  this  aids  both  enhanced  detection 
and  enhanced  recognition  capabilities. 

Figure  14  shows  an  intensity  image  of  a  camouflaged  HMMWV  partly  occluded  by  a 
bush.  The  HMMWV  is  barely  detectable  in  this  image  even  to  human  vision.  While  the 
color  and  texture  appearance  of  the  Army  surplus  camouflage  net  concealing  the  HMMWV 
resembles  that  of  the  bush  and  background  terrain,  the  actual  material  from  which  this  net 
is  constructed  can  be  distinguished  by  its  high  degree  of  polarization  reflectivity  as  shown 
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in  Figure  15  with  color  label  Magenta  (i.e.,  segmentation  obtained  by  thresholding  above 
30%  partial  polarization).  This  is  one  demonstration  of  enhanced  target  detection  capability 
using  polarization  vision. 

Figures  16  and  17  both  show  a  HMMWV  vehicle  viewed  about  45  degrees  from  a  frontal 
view  against  a  background  of  prairie  grass  (sun  is  oriented  directly  in  front  of  the  vehicle 
at  about  a  45  degree  incident  slant).  Figure  16  is  an  intensity  image,  and  Figure  17  is  a 
polarization  image  of  the  same  scene.  Figure  17  shows  a  variety  of  polarization  signatures 
occurring  for  the  HMMWV,  directly  related  to  both  material  composition  of  parts  of  the  ve¬ 
hicle,  and  orientation  of  these  parts  of  the  vehicle.  The  anti-ballistic  glass  on  the  HMMWV 
(as  well  as  on  other  military  vehicles)  has  particularly  salient  polarization  properties  with 
respect  to  light  passing  through  the  glass,  and,  light  specularly  reflecting  off  the  glass.  The 
portions  of  anti-ballistic  glass  at  the  left  of  the  vehicle  in  Figure  17  shows  initially  unpo¬ 
larized  light  from  the  prairie  grass  being  polarized  as  it  passes  through  the  glass.  Because 
the  anti-ballistic  glass  is  non-uniformly  birefringent  across  the  glass,  different  orientations  of 
partial  linear  polarization  produce  a  rainbow  color  pattern  of  lines  using  color  visualization. 
Specular  reflection  from  the  anti-ballistic  glass  windows  on  the  right  of  the  HMMWV  pro¬ 
duce  signiflcant  partial  polarization,  phase  of  this  polarization  (seen  in  blue)  related  to  the 
orientation  of  the  glass.  Cyan  and  green  color  hues  at  the  front  of  the  HMMWV  vehicle  are 
primarily  from  rubber  and  other  dielectric  material  comprising  the  tires,  front  fender,  and, 
the  radiator  grill.  In  Figure  17  painted  surfaces  with  surface  normal  oriented  significantly 
away  from  the  midway  direction  between  the  viewing  and  solar  illumination  directions  have 
relatively  insignificant  partial  polarization.  Note  that  the  painted  frames  around  the  front 
windshields  of  the  HMMWV  with  surface  normal  oriented  not  too  far  from  midway  between 
the  viewing  and  solar  illumination  directions,  have  signiflcant  partial  polarization.  Another 
salient  polarization  signature  comes  from  the  parabolic  reflectors  of  the  headlights  producing 
a  circular  rainbow  in  color  visualization  which  is  visible  even  with  few  pixels  making  up  the 
headlights.  (The  color  near  the  back  right  wheel  in  Figure  17  is  due  to  grass  swaying  in  the 
wind  and  is  not  a  real  polarization  effect.) 

1.4.3  Automated  Assessment  of  Damage  to  Ship  Hulls 

A  very  important  task  for  the  U.S.  Coast  Guard  is  the  rapid  assessment  of  damage  to  a  ship 
that  has  run  aground  (e.g.,  the  Exxon  Valdez)  to  evaluate  how  to  minimize  further  damage 
to  the  ship  and  to  the  surrounding  environment.  Another  important  task  is  the  thorough 
inspection  of  the  hulls  of  ships  in  dock  for  routine  maintenance,  identifying  damage  such  as 
from  scrapes  and  corrosion  requiring  repair.  Currently  the  U.S.  Coast  Guard  uses  frogmen 
divers  to  inspect  portions  of  ship  hulls  underwater,  a  task  that  can  be  very  dangerous  as  well 
as  having  limited  inspection  access.  Under  consideration  is  the  development  of  an  underwater 
Remote  Operated  Vehicle  (ROV)  using  sensors  to  automatically  gather  data  from  the  ship’s 
hull  to  assess  damage.  Under  sponsorship  of  the  U.S.  Coast  Guard,  one  of  the  sensors 
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currently  being  evaluated  for  use  on  such  an  ROV  is  a  Liquid  Crystal  Polarization  Camera 
because  of  its  enhanced  ability  to  detect  exposed  metal,  and  rust. 

Figures  18  thru  21  show  applications  to  Ship  Hull  Damage  Assessment  using  the  Polar¬ 
ization  Camera  in  Figure  5.  The  intensity  images  in  Figures  18  and  20  are  of  a  steel  panel 
that  has  been  painted  with  a  white  coat  of  diffuse  paint,  and  then  an  orange  coat  of  diffuse 
paint.  This  application  of  Polarization  Vision  exploits  how  materials  depolarize  incident 
linear  polarized  light  upon  reflection.  A  polarization  model  predicting  reflected  polarization 
from  materials  as  a  function  of  any  incident  polarization  state  of  light  is  presented  in  [14]. 
The  ROV  would  provide  its  own  illumination,  in  this  case  with  a  linear  polarizer  mounted 
on  a  light  source.  In  Figures  18  thru  21  the  surface  normal  of  the  steel  panel  in  each  case  is 
parallel  to  the  optic  axis  of  the  camera,  and  the  illuminator  is  oriented  parallel  to  viewing. 
Incident  linear  polarized  light  is  oriented  parallel  in  the  images,  this  orientation  really  not 
mattering  due  to  normal  incidence  of  light. 

In  Figure  18  scrape  damage  has  occurred  on  the  panel,  some  scrapes  only  exposing  the 
first  layer  of  white  paint,  other  scrapes  penetrating  all  the  way  down  to  the  steel  metal.  From 
the  intensity  image  in  Figure  18  it  is  difficult  to  tell  which  scrapes  reveal  only  white  paint 
and  where  bare  metal  is  exposed.  Reflected  intensity  and  color  look  the  same  from  these  two 
types  of  damage.  Diffuse  reflection  from  paint  will  significantly  depolarize  incident  linear 
polarized  light,  while  linear  polarization  reflected  from  exposed  metal  will  remain  almost 
completely  linearly  polarized,  especially  at  normal  incidence.  Reflected  polarization  from 
metal  has  partial  polarization  of  nearly  1.0,  while  partial  polarization  from  bright  white 
paint  is  below  0.1.  Diffuse  reflection  from  orange  paint  has  a  partial  polarization  of  about 
0.3  .  Hence,  using  partial  polarization,  scrape  damage  can  be  robustly  distinguished.  In 
Figure  19  Magenta  superimposed  label  depicts  scrape  damage  down  to  white  paint  layer, 
Green  superimposed  label  depicts  scrape  damage  down  to  steel  metal.  It  is  very  important 
to  identify  where  metal  has  been  exposed  from  damage  because  of  rust  and  corrosion  that 
will  soon  occur.  Color  analysis  cannot  distinguish  white  paint  from  exposed  metal  under 
most  conditions  since  they  reflect  the  same  color  of  light  from  the  illuminator. 

Figure  20  shows  another  part  of  the  panel  where  scrape  damage  has  occurred  some  of 
this  exposing  steel  metal  which  has  subsequently  rusted.  A  good  portion  of  this  rust  has 
leaked  out  over  undamaged  portions  of  the  panel,  but  this  is  hard  to  automatically  tell  from 
intensity  or  color.  Note  that  the  intensity  of  rust  at  the  top  of  Figure  20  is  the  same  intensity 
as  that  of  scrapes  downto  the  white  painted  layer  so  that  intensity  properties  are  not  enough 
to  distinguish  phenomena  in  this  image.  Also  rust  has  the  same  color  as  the  standard  orange 
painted  surface  of  the  steel  panel.  Rust  as  it  turns  out  has  a  significantly  higher  specular 
albedo  than  most  diffuse  paints,  and  as  a  result  does  not  depolarize  linearly  polarized  light 
nearly  as  much.  In  Figure  20  rusted  portions  of  the  panel  have  partial  polarization  0.7- 
0.8  which  readily  distinguishes  it  from  undamaged  portions  and  scrape  damage  down  to 
the  first  paint  layer.  In  Figure  21  Yellow  superimposed  label  depicts  rust  and  Magenta 
superimposed  label  depicts  scrape  damage  down  to  white  paint  layer.  Rusted  metal  can 


12 


also  be  distinguished  from  recently  exposed  bare  (i.e.,  unrusted)  metal  as  rust  does  slightly 
depolarize  incident  linearly  polarized  light. 


1.4.4  Marine  Biology 

It  has  been  discovered  over  the  past  two  decades  that  a  number  of  underwater  marine 
animals,  including  fish  and  shrimp,  have  significant  Polarization  Vision  [15],  [12],  [16],  yet  the 
behavioral  significance  of  this  visual  sensory  modality  remains  a  mystery.  The  Liquid  Crystal 
Polarization  Camera  is  providing  the  first  comprehensive  look  of  what  the  underwater  world 
looks  like  in  the  domain  of  Polarization  Vision,  giving  important  insight  into  how  many 
underwater  animals  may  “see.”  Marine  Biologists  at  University  of  Maryland-  Baltimore 
County  are  studying  underwater  polarization  vision  using  Polarization  Camera  Technology 
developed  under  ARPA  funding. 

Upon  Snell  refraction  into  ocean  water  sunlight  becomes  slightly  polarized,  but  just  as 
with  polarization  of  skylight  most  polarization  of  light  in  ocean  water  arises  from  scattering 
[11].  In  the  case  of  ocean  water  the  scatterers  are  suspended  particles  ranging  from  minerals 
to  microscopic  organisms  resulting  in  partial  linear  polarization  which  is  oriented  primarily 
horizontal  with  respect  to  the  ocean  surface  and  having  partial  polarization  up  to  as  much 
as  50  —  60%  depending  upon  the  density  and  types  of  suspended  scatterers.  Most  marine 
animals  underwater  appear  to  reflect  unpolarized  light.  To  a  biological  or  artificial  sensor 
having  Polarization  Vision  this  can  provide  an  advantage  with  respect  to  enhanced  contrast  of 
the  unpolarized  object  against  the  significantly  polarized  background  of  seawater.  Intensity 
and  color  vision  become  blurred  the  further  objects  are  away  from  the  viewer. 

Figure  22  shows  an  example  of  a  color  image  of  fish  underwater,  their  outlines  hard  to 
discern  due  to  the  fact  that  they  reflect  back  the  same  color  of  seawater,  and  that  the  seawater 
is  a  bit  “murky”  from  suspended  particles.  Figure  23  shows  a  partial  linear  polarization  image 
of  the  same  scene.  The  reddish  color  hue  of  seawater  showing  nearly  horizontally  oriented 
partial  linear  polarization  while  the  fish  themselves  are  almost  completely  achromatic.  This 
illustrates  how  polarization  vision  may  be  used  for  contrast  enhancement  in  biological  vision 
systems,  which  may  be  exploited  for  enhanced  underwater  automatic  target  detection  and 
recognition  in  certain  cases.  Note  how  the  two  small  fish  in  Figure  22  (one  near  the  top,  and 
another  to  the  right  in  the  image)  are  almost  completely  invisible,  while  in  the  polarization 
in  Figure  23  they  are  more  apparent.  The  other  fish  have  enhanced  contrast  as  well.  Apart 
from  Image  Understanding  applications,  a  polarization  camera  can  enhance  human  vision 
underwater. 

Polarization  cameras  are  revealing  that  some  portions  of  certain  marine  animals  reflect 
polarized  light,  which  may  be  important  to  explaining  some  behavioral  characteristics  of 
these  animals.  Figure  24  shows  a  hue-saturation-intensity  representation  of  a  partial  linear 
polarization  image  of  a  Mantis  Shrimp  (lying  upside-down).  Of  interest  are  the  polarizing 
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properties  of  the  frontal  antennal  scales  at  the  right  of  this  image-  in  the  partial  linear 
polarization  image  one  scale  appears  in  color  hue  purple,  the  other  scale  appears  in  color 
hue  yellow  indicating  almost  a  90'’  difference  in  orientation  of  polarization  between  the  two 
scales.  Before  such  polarization  images  taken  with  a  Polarization  Camera  this  potentially 
important  characteristic  was  not  known.  The  origin  of  polarization  is  not  from  specular  or 
diffuse  reflection-it  turns  out  that  these  antennal  scales  consist  of  organic  micropolarizers 
that  are  built  into  the  structure  of  the  animal  itself!  The  Mantis  Shrimp  which  possesses 
polarization  vision  can  vary  this  polarization  orientation  difference  by  moving  its  frontal 
scales  and  one  conjecture  is  that  this  can  possibly  be  used  as  a  method  of  communication 
or  identification.  Other  significant  polarization  effects  are  being  noticed  with  starfish,  and 
lionfish  which  may  also  have  similar  polarizing  microstructures.  Polarization  cameras  may 
reveal  a  potentially  whole  new  area  in  Marine  Biology. 


1.5  Conclusions  and  Future  Directions  For  Polarization  Vision 

Polarization  Vision  is  a  new  sensory  modality  for  image  understanding  that  can  significantly 
augment  the  capabilities  of  computer  vision  systems,  as  well  as  expand  the  horizons  of 
human  visual  awareness.  This  is  becoming  more  evident  as  Polarization  Camera  Technology 
matures  and  becomes  readily  available  to  a  diverse  group  of  researchers  for  use  outdoors 
and  underwater.  A  broad  overview  of  recent  results  was  presented  over  a  diverse  set  of 
applications  where  either  basic  low-level  processing,  or,  a  natural  visualization  of  polarization 
data  revealed  important  scene  information  that  was  not  as  evident  from  intensity  or  color. 

It  is  felt  that  Polarization  Vision  has  not  yet  nearly  reached  its  full  role  in  Computer 
Vision.  The  full  application  potential  of  Polarization  Vision  will  be  revealed  as  there  is 
wider  use  of  Polarization  Cameras  in  various  research  science  areas.  While  Liquid  Crystal 
Polarization  Cameras  are  proving  to  be  quite  a  practical  tool  for  studying  Polarization  Vision 
on  their  own,  developments  are  already  underway  for  the  next  generation  of  polarization 
cameras  utilizing  more  self-contained  technology  that  will  eventually  produce  very  high- 
resolution  polarization  images  at  up  to  1000  images  a  second. 


2  Other  Research  Efforts 

2.1  Diffuse  Reflection  Model 

One  of  the  most  common  assumptions  for  recovering  object  features  in  computer  vision  and 
rendering  objects  in  computer  graphics  is  that  diffuse  reflection  from  materials  is  Lamber¬ 
tian.  A  limitation  of  this  work  is  that  there  is  no  account  for  the  behavior  of  diffuse  reflection 
in  the  limit  as  rough  surfaces  become  mildly  rough  or  smooth-  an  important  and  large  class 
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of  surfaces  in  practice.  This  paper  shows  that  there  is  significant  deviation  from  Lamber¬ 
tian  behavior  in  diffuse  reflection  from  smooth  surfaces  not  predicted  by  existing  reflectance 
models,  having  an  important  bearing  on  any  computer  vision  technique  that  may  utilize 
reflectance  models  including  shape-from-shading  and  binocular  stereo.  In  particular,  con¬ 
trary  to  prediction  by  Lambert’s  Law,  diffuse  reflection  from  smooth  surfaces  is  significantly 
viewpoint  dependent,  and  there  are  prominent  diffuse  reflection  maxima  effects  occurring  on 
objects  when  incident  point  source  illumination  is  greater  than  50°  relative  to  viewing  in¬ 
cluding  the  range  from  90°  to  180°  where  the  light  source  is  behind  the  object  with  respect  to 
viewing.  We  present  here  a  diffuse  reflectance  model,  derived  from  first  physical  principles, 
that  accurately  predicts  diffuse  reflection  from  smooth  inhomogeneous  dielectric  surfaces  as  a 
function  of  both  viewing  angle,  and,  angle  of  incidence.  Utilizing  results  of  radiative  transfer 
theory  for  subsurface  multiple  scattering  this  new  model  precisely  accounts  for  how  incident 
light  and  the  distribution  of  subsurface  scattered  light  is  influenced  by  Fresnel  attenuation 
and  Snell  refraction  at  a  smooth  air-dielectric  surface  boundary.  A  number  of  experimental 
results  have  verified  the  proposed  diffuse  reflectance  model. 

A  prevalent  class  of  materials  encountered  both  in  common  experience  and  in  computer 
vision/robotics  environments  are  inhomogeneous  dielectrics  which  include  plastics,  ceramics, 
and,  rubber.  In  computer  vision  a  widely  used  assumption  about  diffuse  reflection  from 
materials  is  Lambert’s  law  [17],  namely  the  expression: 

—Lp  cos  tpduj 

TC 

where  p  is  termed  the  diffuse  albedo  in  the  range  [0, 1.0].  This  reflectance  model  is  typically 
instantiated  into  the  implementation  of  a  large  number  of  algorithms  such  as  shape-from- 
shading  [18]  and  photometric-based  binocular  stereo  [19],  [20].  It  is  therefore  important  for 
researchers  in  the  computer  vision  community  who  utilize  assumptions  about  diffuse  reflec¬ 
tion  to  be  aware  of  the  conditions  under  which  there  is  significant  deviation  from  Lambert’s 
law. 

Almost  all  diffuse  reflection  from  inhomogeneous  dielectrics  physically  arises  from  subsur¬ 
face  multiple  scattering  of  light  caused  by  subsurface  inhomogeneities  in  index  of  refraction. 
In  this  paper  we  model  inhomogeneous  dielectric  material  as  a  collection  of  scatterers  con¬ 
tained  in  a  uniform  dielectric  medium  with  index  of  refraction  different  from  that  of  air.  We 
derive  an  expression  for  diffuse  reflected  radiance  resulting  from  the  process  of  incident  light 
refracting  into  the  dielectric  medium  across  a  smooth  surface  boundary,  producing  a  sub¬ 
surface  diffuse  intensity  distribution  from  multiple  internal  scattering,  and  then  refraction  of 
this  subsurface  diffuse  intensity  distribution  back  out  into  air.  We  also  account  for  the  infi¬ 
nite  progression  of  internal  specular  reflection  at  the  air-dielectric  boundary  and  sub-surface 
scattering.  A  common  property  of  diffuse  reflection  from  smooth  inhomogeneous  dielectric 
surfaces  is  that  such  reflection  is  azimuth  independent  with  respect  to  viewing  about  the 
surface  normal,  regardless  of  the  fixed  direction  of  incident  light.  We  have  formally  derived 
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and  empirically  verified  that  for  smooth  inhomogeneous  dielectric  surfaces  that  exhibit  such 
azimuth  symmetric  diffuse  reflection,  if  light  is  incident  with  radiance,  L,  at  incidence  angle, 
^0,  through  a  small  solid  angle,  do;,  then 


X  (1  —  F('i/^,n))  X  cost/;  X  (1  —  F(sin  ^{-—^),\/n))duj  (4) 

describes  the  reflected  radiance  into  viewing  angle,  </>,  (i.e.,  angle  between  viewing  and  the 
surface  normal,  also  known  as  emittance  angle)  [21],  [22],  The  terms  F{, )  refer  to  the  Fresnel 
reflection  coefficients  [23],  n,  is  the  index  of  refraction  of  the  dielectric  medium,  and,  g,  is  the 
total  dijfuse  albedo.  We  showed  that  the  total  diffuse  albedo,  g,  is  directly  related  to  both  the 
single  scattering  albedo  describing  the  proportion  of  energy  reradiated  upon  each  subsurface 
single  scattering,  and,  the  index  of  refraction  n.  This  current  paper  also  presents  a  much 
more  comprehensive  set  of  various  results  demonstrating  non-Lambertian  diffuse  reflection 
effects  from  smooth  surfaces  explained  by  expression  4. 

Particularly  useful  to  object  feature  extraction  in  computer  vision,  our  expression  4  for 
diffuse  reflection  allows  precise  characterization  of  the  conditions  under  which  the  Lambertian 
model  breaks  down  for  inhomogeneous  dielectrics  and  where  our  more  accurate  model  should 
be  used.  We  show  that  Lambert’s  law  is  valid  for  smooth  dielectrics  to  within  5%  only  as 
long  as  both  angle  of  incidence,  and  viewing  angle,  (/>,  are  simultaneously  less  than  50'’. 
This  means  that  for  applications  in  computer  vision  there  are  a  large  number  of  situations  in 
which  Lambert’s  law  is  significantly  in  error-  near  the  occluding  contour  of  objects  under  any 
illumination  condition;  for  illuminations  incident  at  greater  than  50°  relative  to  viewing  there 
will  be  significant  errors  both  near  the  occluding  contour  and  over  a  large  portion  of  object 
area  bounded  on  one  side  by  the  shadow  boundary  with  respect  to  illumination;  for  multiple 
images  of  a  smooth  dielectric  object  there  will  be  significant  viewpoint  dependence  of  diffuse 
reflection  for  most  object- viewpoint  situations.  Existing  diffuse  reflectance  models  do  not 
account  for  these  errors.  In  addition  to  experimentally  verifying  the  viewpoint  dependence 
of  diffuse  reflection,  we  show  how  the  diffuse  reflection  maximum  occurring  between  an 
occluding  contour  and  the  shadow  boundary  for  a  smooth  dielectric  illuminated  by  a  point 
light  source  is  accurately  predicted  because  of  this  viewpoint  dependence.  For  illumination 
incident  at  90°  and  greater  relative  to  viewing,  a  diffuse  reflection  maximum  is  not  even 
predicted  by  existing  models  having  dependence  only  on  angle  of  incidence,  when  in  fact  this 
maximum  is  very  prominent.  Furthermore,  diffuse  reflection  is  empirically  observed  to  be 
very  small  in  the  immediate  vicinity  of  an  occluding  contour  for  all  illumination  conditions,  an 
observation  that  is  not  supported  by  existing  models,  and  yet  accounted  for  by  expression  4. 


2.2  Relative  Strength  of  Diffuse  and  Specular  Reflection 

An  important  feature  of  our  diflfuse  reflectance  model  is  that  it  predicts  the  surface  diffuse 
albedo  for  dielectrics  purely  in  terms  of  physical  parameters.  Inhomogeneous  dielectric  sur- 
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faces  exhibit  both  diffuse  and  specular  reflection  components.  Previously,  combined  diffuse 
and  specular  reflection  has  been  modeled  as  a  sum  of  scaled  diffuse  and  specular  terms,  with 
the  scaling  factors  (i.e.,  diffuse  albedo  and  specular  albedo)  determined  from  experimental 
fitting  for  each  particular  surface.  We  propose  the  following  combined  reflectance  model  for 
diffuse  and  specular  reflected  radiance  from  smooth  surfaces  [24]: 

Lq[1-  F{ip,n)]  cos'i/;[l  -  F{sm^^{^~),l/n)]duj 


+  L  Ffip,  n)  6{iP  -  f)  5(00  +  180  -  0),  (5) 

for  small  incident  solid  angle,  do;,  at  incidence  angle  ip,  incident  azimuth  angle  0o,  emittance 
angle  <p  and  emitted  azimuth  angle  0. 

The  combined  diffuse  and  specular  reflectance  model,  expression  17  can  be  used  to  for¬ 
mally  answer  the  question:  How  bright  is  a  specularity  ?  Taking  the  ratio  of  the  strengths 
of  the  specular  to  diffuse  reflection  components  of  expression  5: 

_ F{i^,n)  _  .g. 

p  [1  —  F{'ip,  n)]  008-0  [1  —  F{sm~^{^^),  l/n)]  duj 

which  is  independent  of  uniform  incident  radiance,  L.  Since  F{x,n)  and  F{x,l/n)  are 
monotonically  increasing  with  respect  to  increasing  x  it  should  be  clear  that  diffuse  reflection 
is  at  a  maximum  when  xp  =  0°,  0  =  0°  and  specular  reflection  is  at  a  minimum  when  -0  =  0°. 
Therefore,  the  ratio  expression  6  is  a  minimum  when  viewing-illumination  geometry  is  at 
normal  incidence  and  viewing.  The  following  ratio  constitutes  a  physical  lower  bound  on  the 
ratio  of  specular  to  diffuse  reflected  radiance: 


_ F{0^,ri) _ 

p  [1  -  F(0°,  n)] [1  -  F(0°,  l/n)]  duj  ' 

The  total  diffuse  albedo,  p,  is  largest  for  conservative  scattering  when  the  single  scattering 
albedo  p  =  1.0.  Using  expression  16  to  compute  p  at  p  =  1.0,  the  physical  lower  bound 
expressed  by  7  for  the  ratio  of  specular  to  diffuse  reflected  radiance  evaluated  for  different 
indices  of  refraction  is  {duj  in  steradians): 


(n  =  1.4) 


0.0838 

du 


{n  =  1.7) 


0.211 

duj 


in 


2.0) 


0.370 

dto 


These  expressions  represent  physical  lower  bounds  below  which  it  is  not  physically  possi¬ 
ble  to  have  brightness  contrast  between  a  specularity  and  surrounding  diffuse  reflection  on  a 
smooth  dielectric  surface.  Since  practically  all  dielectrics  have  n  >  1.4,  this  particular  lower 
bound  can  serve  as  a  very  conservative  general  formula  for  ruling  out  specularities  in  a  scene. 
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2.3  3-D  Stereo 


We  have  developed  a  novel  robust  methodology  for  corresponding  a  dense  set  of  points  on 
an  object  surface  from  photometric  values,  for  3-D  stereo  computation  of  depth  [25].  The 
methodology  utilizes  multiple  stereo  pairs  of  images,  each  stereo  pair  taken  of  exactly  the 
same  scene  but  under  different  illumination.  With  just  2  stereo  pairs  of  images  taken  respec¬ 
tively  for  2  different  illumination  conditions,  a  stereo  pair  of  ratio  images  can  be  produced; 
one  for  the  ratio  of  left  images,  and  one  for  the  ratio  of  right  images.  We  demonstrate  how 
the  photometric  ratios  composing  these  images  can  be  used  for  accurate  correspondence  of 
object  points.  Object  points  having  the  same  photometric  ratio  with  respect  to  2  differ¬ 
ent  illumination  conditions  comprise  a  well-defined  equivalence  class  of  physical  constraints 
defined  by  local  surface  orientation  relative  to  illumination  conditions.  We  formally  show 
that  for  diffuse  reflection  the  photometric  ratio  is  invariant  to  varying  camera  characteristics, 
surface  albedo,  and  viewpoint  and  that  therefore  the  same  photometric  ratio  in  both  images 
of  a  stereo  pair  implies  the  same  equivalence  class  of  physical  constraints.  Corresponding 
photometric  ratios  along  epipolar  lines  in  a  stereo  pair  of  images  under  different  illumina¬ 
tion  conditions  is  therefore  a  robust  correspondence  of  equivalent  physical  constraints,  and 
determination  of  depth  from  stereo  can  be  performed  without  explicitly  knowing  what  these 
physical  constraints  being  corresponded  actually  are.  While  illumination  planning  is  required 
our  photometric-based  stereo  methodology  does  not  require  knowledge  of  illumination  con¬ 
ditions  in  the  actual  computation  of  3-D  depth,  and  is  applicable  to  perspective  views.  This 
technique  extends  stereo  determination  of  3-D  depth  to  smooth  featureless  surfaces  without 
the  use  of  precisely  calibrated  lighting.  We  demonstrate  experimental  depth  maps  from  a 
dense  set  of  points  on  smooth  objects  of  known  ground  truth  shape,  determined  to  within 
1%  depth  accuracy. 

To  describe  the  problematic  issues  of  comparing  image  intensities  between  a  stereo  pair 
of  cameras  we  need  to  understand  about  the  image  formation  process  and  the  nature  of 
reflection  from  objects.  We  describe  the  formation  of  image  intensity  values  beginning  with 
the  familiar  relation  from  Horn  and  Sjoberg  [26]: 

E  =  Lr(7r/4)(D/?)^cos^  a  ,  (8) 

relating  image  irradiance,  E,  to  reflected  radiance,  Lr-  The  lens  diameter,  D,  image  distance, 
i,  and  light  angle,  a,  incident  on  the  camera  lens.  Equation  8  assumes  ideal  pinhole  optics. 
The  effective  diameter,  D,  of  a  lens  can  be  controlled  with  an  aperture  iris  the  size  of  which 
is  measured  on  an  F-stop  scale.  Image  irradiance  is  very  sensitive  to  F-stop.  While  a  stereo 
pair  of  cameras  can  use  identical  model  lenses  at  exactly  the  same  F-stop  setting,  the  effective 
lens  diameters  can  still  be  slightly  different.  The  focal  lengths  as  well  can  be  slightly  different 
and  by  the  classical  thin  lens  law  this  will  influence  the  image  distance,  i,  in  turn  effecting 
the  image  irradiance.  Even  in  the  ideal  case  where  focal  lengths  are  precisely  equal,  the 
image  distance,  i,  can  be  slightly  different  for  a  stereo  pair  of  images  even  though  the  images 
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“appear”  equivalently  in  focus  when  in  fact  they  are  not  precisely  in  focus.  On  top  of  all 
this  is  the  dependence  of  image  irradiance  in  perspective  images  on  pixel  location  relative  to 
the  optical  center  of  the  image  plane.  The  farther  a  pixel  is  radially  away  from  the  optical 
center,  the  larger  is  the  light  incident  angle,  a,  which  strongly  effects  image  irradiance.  Image 
irradiances  arising  from  the  same  object  point  appear  in  different  parts  of  a  stereo  pair  of 
images  making  them  difficult  to  compare. 

Equation  8  only  takes  into  account  the  optics  involved  in  image  formation.  Image  irradi¬ 
ance  is  converted  into  pixel  gray  value  using  electronics.  In  general  the  conversion  of  image 
irradiance,  E,  into  pixel  gray  value,  /,  can  be  described  by  the  expression 

I  =  gE^  +  d,  (9) 

where  g  is  termed  the  gain,  d  is  the  dark  reference,  and,  7,  controls  the  non-linearity  of  gray 
level  contrast.  It  is  typically  easy  to  set  7  =  1.0  producing  a  linear  response,  and  easy  to  take 
a  dark  reference  image  with  the  lens  cap  on,  then  subtracting  d  out  from  captured  images. 
However,  we  have  observed  that  not  only  can  the  gain,  g,  be  variable  between  identical  model 
cameras  but  this  can  change  over  time  especially  for  relatively  small  changes  in  temperature. 
Unless  g  is  calibrated  frequently,  comparing  pixel  gray  values  for  identical  image  irradiances 
between  a  stereo  pair  of  cameras  can  be  difficult. 

A  widely  used  assumption  about  diffuse  reflection  from  materials  is  that  they  are  Lam¬ 
bertian  [17]  meaning  that  light  radiance,  L,  incident  through  solid  angle,  du,  at  angle  of 
incidence,  ip,  produces  reflected  radiance: 

L  p  cos  Ip  duj 

independent  of  viewing  angle.  The  independence  of  diffuse  reflected  radiance  with  respect  to 
viewing  angle  makes  it  theoretically  feasible  to  associate  radiance  values  with  object  points 
in  a  stereo  pair  of  images.  However,  even  for  ideal  Lambertian  diffuse  reflectance  the  above 
discussion  outlines  the  practical  difficulties  in  achieving  an  accurate  correspondence  of  pixel 
gray  values  produced  from  reflected  radiance  in  a  stereo  pair  of  cameras. 

The  physical  reality  of  diffuse  reflection  makes  it  even  more  practically  difficult  to  asso¬ 
ciate  diffuse  reflected  radiance  with  object  points  across  a  stereo  pair  of  images.  A  recently 
proposed  diffuse  reflectance  model  for  smooth  dielectric  surfaces  [21],  [27],  [22]  empirically 
verified  to  be  more  accurate  than  Lambert’s  Law,  expresses  the  dependence  of  diffuse  re¬ 
flected  radiance  on  both  angle  of  incidence,  ip,  and  viewer  angle,  (p,  as 

Lp[l  -F{ip,n)]  cosV'[l  -  F(sin“^(^^),  1/n)]  do;  (10) 

Tt 

where  the  functions,  F{),  are  the  Fresnel  reflection  coefficients  [23],  and,  n,  is  the  index  of 
refraction  of  the  dielectric  surface,  and,  p,  is  the  diffuse  albedo.  Diffuse  reflected  radiance 
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from  an  objact  point  as  sean  from  the  two  different  viewpoints  of  a  stereo  pair  of  cameras 
will  almost  always  not  be  equal. 

Combining  equations  8,  9  and  10  gives  us  an  expression  which  precisely  relates  pixel  gray 
value,  I,  to  diffuse  reflection  as  a  function  of  imaging  geometry  and  camera  parameters.  For 
incident  radiance  L  through  a  small  solid  angle,  du>,  at  an  angle  of  incidence,  ip,  the  gray 
value  formed  from  viewing  angle,  (p  (assuming  we  subtract  out  the  dark  reference,  d)  is: 


OG  Y)  (i) 

I  =  g[{7r/4){D/if  cos"^  a  Lp[l- F{ip,n)]  x  cosip  x  [1  -  F(sin"^(-^),  1/n)]  dw 

For  a  general  incident  radiance  distribution,  L{ip,6),  {9  is  azimuth  about  the  surface 
normal)  with  respect  to  an  object  point,  integrating  over  the  incident  hemisphere,  the  gray 
value  will  be: 


=  g[{7r/4){D/i)^cos^a  j  L{iP,  9)  p  [1- F{iP,n)]  x  cos  ip x  [1-F (sin  i(^),  1/n)]  do; 


=  g[{7T/4)iD/ifcos^a[l-F{sm-\—),l/n)]pY^'^x[  f  L{ip,9)[l-F{ip,n)]  cosipdu 

”  (11) 
Consider  this  object  point  first  illuminated  with  an  incident  radiance  distribution,  Li{ip,9), 
and  then  illuminated  with  an  incident  radiance  distribution,  L2{ip,9).  From  equation  11  the 
photometric  ratio  of  gray  values  is: 

h  [  JLi{ip,9)[l- F {ip,  n)]  cos  Ip  duj  .^2) 

h  ~~  [  !  L2{ip,  9)  [1- F {ip,  n)]  cos  Ip  d(j 

where  the  terms  outside  the  integral  signs  in  the  numerator  and  the  denominator  cancel 
out.  This  photometric  ratio  expresses  a  well-defined  physical  constraint  determined  by  an 
illumination  distribution  relative  to  surface  orientation  at  an  object  point.  It  is  a  function 
of  only  incident  light  geometry  relative  to  local  surface  orientation,  and  is  invariant  to  all 
camera  parameters  (except  7),  viewing  angle,  <p,  and  diffuse  surface  albedo,  p.  Note  that 
this  photometric  ratio  is  local  to  an  object  surface  point,  so  that  it  does  not  matter  if  the 
incident  lighting  distribution  changes  from  object  point  to  object  point  (i.e.,  which  can  occur 
if  lighting  elements  are  close  to  the  object  relative  to  object  size).  This  photometric  ratio  is 
an  invariant  with  respect  to  each  object  point. 
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2.4  Segmenting  Curvature 

Gaussian  curvature  is  an  intrinsic  local  shape  characteristic  of  a  smooth  object  surface  that  is 
invariant  to  orientation  of  the  object  in  3-D  space  and  viewpoint.  Accurate  determination  of 
the  sign  of  Gaussian  curvature  at  each  point  on  a  smooth  object  surface  (i.e.,  the  identification 
of  hyperbolic,  elliptical  and  parabolic  points)  can  provide  very  important  information  for  both 
recognition  of  objects  in  automated  vision  tasks  and  manipulation  of  objects  by  a  robot.  We 
present  a  multiple  illumination  technique  that  directly  identifies  elliptical,  hyperbolic,  and 
parabolic  points  from  diffuse  reflection  from  a  smooth  object  surface.  This  technique  is  based 
upon  a  photometric  invariant  involving  the  behavior  of  the  image  intensity  gradient  under 
varying  illumination  under  the  assumption  of  the  image  irradiance  equation.  The  nature  of 
this  photometric  invariant  allows  direct  segmentation  of  a  smooth  object  surface  according 
to  the  sign  of  Gaussian  curvature  independent  of  knowledge  of  local  surface  orientation, 
independent  of  diffuse  surface  albedo,  and  with  only  approximate  knowledge  of  the  geometry 
of  multiple  incident  illumination.  In  comparison  with  photometric  stereo,  this  new  technique 
determines  the  sign  of  Gaussian  curvature  directly  from  image  features  without  having  to 
derive  local  surface  orientation,  and,  does  not  require  calibration  of  the  reflectance  map  from 
an  object  of  known  shape  of  similar  material  or  precise  knowledge  of  all  incident  illuminations. 
We  demonstrate  how  this  segmentation  technique  works  under  conditions  of  simulated  image 
noise,  and  actual  experimental  imaging  results. 

Horn  and  Sjoberg  [26]  defined  the  image  irradiance  equation  which  equates  image  ir¬ 
radiance,  I{x,y),  to  reflected  radiance,  R{p,q)  up  to  a  multiplicative  constant  involving 
camera  parameters.  Assuming  that  this  constant  factor  is  known  and  incorporated  into  the 
reflectance  map,  then 


Iix,y)  =  R{p,q)  (13) 

which  for  a  known  reflectance  map  allows  the  derivation  of  a  surface  orientation  constraint 
in  terms  of  gradient  variables,  p  and  q  from  the  image  irradiance  at  a  pixel.  In  the  case  of 
the  Lambert  reflectance  map,  expression  9,  the  surface  orientation  constraint  derived  from 
an  image  irradiance  value  is  a  quadratic  expression  in  p  and  q.  Therefore,  the  quantitative 
derivation  of  surface  orientation  from  such  local  information  is  underconstrained  to  an  infinity 
of  possible  orientations  along  a  quadratic  curve  in  gradient  space. 

As  local  surface  orientation  is  related  to  the  first  order  variation  of  the  surface  height 
function,  f{x,y),  local  surface  curvature  is  related  to  the  first  order  variation  of  surface  ori¬ 
entation  (which  in  turn  involves  second  order  variations  in  the  height  function).  Constraints 
on  surface  curvature  can  be  obtained  photometrically  by  equating  the  derivatives  of  the 
image  irradiance  equation  13.  Making  the  following  notational  simplifications 

Ix  =  dl/dx,  Iy  =  dl/dy,  Rp  =  dR/dp,  Rq  =  dR/dq, 
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the  following  matrix  equation  is  obtained  from  the  chain  rule: 

Ix  \  _  (  dp/dx  dq/dx  \  f  Rp  \ 

ly  )  V  ^p/^y  )  \  Rq  ) 

The  matrix  in  equation  14  is  the  viewer  centered  curvature  matrix,  which  is  also  the  Hessian 
matrix  with  respect  to  the  height  function,  f{x,y).  Because  the  second  order  differentiation 
of  f{x,y)  by  x  and  by  y  is  commutative,  the  off-diagonal  components  of  the  viewer  cen¬ 
tered  curvature  matrix,  dp/dy  and  dq/dx,  are  equal.  Hence,  the  viewer  centered  curvature 
matrix  is  equivalent  to  the  Jacobian  of  the  transformation  between  image  coordinates  {x,y) 
and  gradient  coordinates  (p,  q).  When  the  viewer  centered  curvature  matrix  multiplies  an  in¬ 
finitesimal  vector  in  image  coordinates,  the  resulting  vector  is  the  corresponding  infinitesimal 
vector  change  in  surface  orientation  represented  in  gradient  coordinates: 


(  dp/dx  dp/dy  \  (  dx  \  _(  dp  \ 
dq/dx  dq/dy  j  \  dy  )  \  / 

The  viewer  centered  curvature  matrix  therefore  relates  a  change  in  image  coordinates  to 
the  corresponding  change  in  local  surface  orientation  at  a  smooth  surface  point.  From  a 
single  image  the  three  independent  components  of  the  viewer  centered  curvature  matrix 
can  be  constrained  by  two  linear  equations,  produced  from  the  matrix  equation  14,  from 
measurement  of  the  image  intensity  gradient. 

An  object  centered  curvature  matrix  can  be  defined  as  the  matrix  which  multiplies  an  in¬ 
finitesimal  vector  in  the  tangent  plane  at  a  smooth  object  point  to  produce  the  corresponding 
vector  change  in  the  surface  normal.  The  determinant  of  this  object  centered  curvature  ma¬ 
trix  is  the  Gaussian  curvature  at  a  point  on  a  smooth  object  surface  which  can  be  shown 
to  be  a  fundamental  invariant  of  the  surface  with  respect  to  how  it  is  oriented  in  3-D  space 
[28].  When  expressed  in  terms  of  gradient  variables  relative  to  a  set  of  image  coordinates, 
the  Gaussian  curvature  at  a  point  is  given  by: 


(dp/dx)  {dq/dy)  -  (dp/dy)  {dq/dx) 

(1  +  q‘^)‘^ 

It  is  very  important  to  note  that  the  sign  of  the  Gaussian  curvature  is  the  same  as  the  sign 
of  the  determinant  of  the  viewer  centered  curvature  matrix  in  equation  14. 

We  consider  the  extension  of  matrix  equation  14  incorporating  two  reflectance  maps, 
R\p,q),  and  R‘^(p,q): 


( \  \  ( 

V  ly  )  ~  [  9p/9y  dq/dy  )\Rl  Rl)  ■ 


(16) 
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Each  of  the  two  image  intensity  gradients  is  produced  at  an  object  point  respective  to  each 
of  the  reflectance  maps.  Labeling  the  matrices  in  equation  16  respectively  from  left  to  right 
as  A,  B,  and,  C  so  that 


A  =  BC, 

we  will  refer  to  matrix  A  as  the  image  intensity  gradient  matrix,  and  to  matrix  C  as  the 
reflectance  map  gradient  matrix.  It  is  clear  that 


DETERMINANT (^)  =  DETERMINANT(R)  x  DETERMINANT (C)  . 

Therefore,  if  the  determinant  of  the  reflectance  map  gradient  matrix  is  known  for  a  given 
surface  orientation,  then  the  sign  of  the  determinant  of  the  viewer  centered  curvature  matrix 
(and  therefore  the  sign  of  the  Gaussian  curvature)  can  be  derived  from  the  computation 
of  the  determinant  of  the  image  intensity  gradient  matrix.  Suppose  that  two  reflectance 
maps  B}{p,q)  and  E?‘{p,q)  are  produced  from  two  different  incident  illuminations.  If  the 
determinant  of  the  reflectance  map  gradient  matrix  has  the  same  known  sign  for  all  sur¬ 
face  orientations  p  ,  q,  then  the  sign  of  Gaussian  curvature  can  be  determined  from  the 
determinant  of  the  measured  intensity  image  gradient  matrix  independent  of  local  surface 
orientation.  So,  for  example,  if  the  determinant  of  the  reflectance  map  gradient  matrix  were 
consistently  positive  for  all  surface  orientations,  a  positive,  negative,  zero  determinant  for  the 
measured  image  intensity  gradient  matrix  respectively  identifies  a(n)  elliptical,  hyperbolic, 
parabolic  point. 

The  determination  of  the  sign  of  Gaussian  curvature  from  equation  16  has  a  direct  ge¬ 
ometric  interpretation  in  terms  of  the  rotational  change  of  the  image  intensity  gradient 
induced  by  varying  illumination.  The  value  of  the  determinant  of  the  image  intensity  gra¬ 
dient  matrix  is  equal  to  the  z-component  of  the  cross-product  of  the  two  image  intensity 
gradients  as  vectors  in  the  x-y  image  plane.  That  is: 


DETERMINANT  |  1  =  lig  -  Ipl 
X  (lUlo)  =  {o.ajpl  -  ipl), 

where  ’x’  is  the  vector  cross-product  operation.  Therefore 


jlj2 


jlj2 

y  X 


IlUx. 4^0)11  •  Il(^x>-^4o)l|sin0  , 


where  9  is  the  angle  between  the  image  intensity  gradient  vectors.  Hence,  the  sign  of  the 
determinant  of  the  image  intensity  gradient  matrix  is  equal  to  the  sign,  9,  of  the  rotational 
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change  between  the  image  intensity  gradient  vectors  before  and  after  the  change  in  illu¬ 
mination.  For  a  fixed  sign  of  the  determinant  of  the  reflectance  map  gradient  matrix  in 
equation  16,  the  direction  of  rotational  change  of  the  image  intensity  gradient  at  an  object 
point  is  determined  by  the  sign  of  the  Gaussian  curvature  at  that  point.  At  parabolic  points 
the  determinant  of  the  viewer  centered  curvature  matrix  is  zero  and  therefore  there  is  no 
rotational  change. 

As  discussed  in  the  previous  section,  the  Lambert  reflectance  map  (expression  9)  provides 
a  good  approximation  to  the  relationship  between  surface  orientation  and  diffuse  reflection 
from  smooth  and  mildly  rough  dielectric  surfaces  for  incident  point  source  illumination  di¬ 
rections  not  too  oblique  with  respect  to  viewing.  Unfortunately  the  sign  of  the  determinant 
of  the  reflectance  map  gradient  matrix  in  equation  16  for  Lambertian  reflectance  using  any 
pair  of  different  incident  point  source  orientations  is  not  the  same  with  respect  to  all  surface 
orientations.  An  easy  proof  of  this  is  to  see  that  the  determinant  of  the  reflectance  map 
gradient  matrix  for  Lambertian  reflectance  is  zero  for  surface  orientations  p,  q  equal  to  any 
of  the  two  incident  point  source  orientations  that  are  used.  If  it  is  known  a  prion  what 
restricted  range  of  surface  orientations  will  be  examined  it  may  be  possible  to  use  two  in¬ 
cident  point  source  orientations  that  will  produce  a  Lambertian  reflectance  gradient  matrix 
having  determinant  the  same  sign  over  this  range  of  surface  orientations.  Our  goal  however 
is  to  come  up  with  a  general  technique  for  finding  the  sign  of  Gaussian  curvature  completely 
invariant  to  all  surface  orientations. 

A  sub-class  of  reflectance  maps  that  results  in  a  reflectance  map  gradient  matrix  whose 
sign  of  determinant  is  invariant  to  all  surface  orientations  is  part  of  what  we  term  the  class  of 
Ratio- Of- Lambertian  reflectance  maps.  Consider  the  ratio  of  Lambertian  reflectance  maps, 
expression  9,  for  two  incident  point  source  orientations  Ps,  Qs,  and,  Pg,  Qg  resulting  in. 


_  Lq+pps  +  qqs  +  (17) 

L'l+PP's  +  QQs  yi  +  +  (gO" 

The  Ratio-Of-Lambertian  reflectance  map  does  not  physically  exist  for  a  single  image  of 
a  diffuse  reflecting  surface,  but  rather  for  the  ratio  of  two  images  of  a  Lambertian  diffuse 
reflecting  surface  each  image  taken  under  a  different  incident  point  source  illumination.  Note 
that  this  reflectance  map  is  invariant  to  albedo,  p.  It  is  also  invariant  to  perspective  effects 
of  the  radiometry  of  image  formation,  (i.e.,the  cos^a  effect  discussed  by  Horn  and  Sjoberg 
[26]  gets  factored  out  for  each  pixel).  In  the  case  where  p'g  =  g'  =  0,  that  is  one  of  the 
incident  illumination  directions  is  parallel  to  viewing,  the  Ratio-Of-Lambertian  reflectance 
map  becomes 


R{p,(l)  =  ^{^+PPs  +  QQs)\/'^+P‘i  +  Q. 


(18) 


which  is  completely  linear  in  gradient  variables  p  and  g.  As  a  height  function  above  the  p  q 
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plane  the  sub-class  of  reflectance  maps  represented  by  expression  18  are  ramp  functions  with 
the  gradient,  {Rp,  Rq),  of  the  ramp  equal  to  a  constant  scalar  multiple  of  the  vector  {ps,qs)- 

From  expression  10,  in  actuality  the  Ratio-Of-Lambertian  reflectance  map  of  expres¬ 
sion  18  is  multiplied  by  the  ratio 


1  -F(^,n) 
l-F(t/;',n)’ 

where  is  the  angle  of  incidence  with  respect  to  the  illumination  parallel  to  viewing,  and 
ifj  is  the  angle  of  incidence  with  respect  to  the  other  light  source.  Notice  that  the  viewing 
dependence,  (f),  completely  cancels  out.  It  turns  out  that  the  ratio  in  expression  19  is  nearly 
constant  at  1.0  for  all  surface  orientations  within  the  wide  range  of  75°  relative  to  viewing. 

Consider  taking  images  of  a  diffuse  reflecting  object  with  respect  to  incident  point  source 
orientations,  (pi,gi),  and,  (p2)92)-  Each  of  these  images  are  then  divided  by  a  third  im¬ 
age  taken  for  incident  illumination  parallel  to  viewing.  Utilizing  the  Ratio-Of-Lambertian 
reflectance  map,  expression  18,  the  reflectance  map  gradient  matrix  is 

n  ^  +  ltP2yi+pl  +  92  \  .20) 

\  +Pl+  Ql  +P2  +  q2  ) 

where  Li,  L2,  and,  Lz  are  the  incident  radiances  of  the  first,  second,  and,  third  point  light 
sources  respectively.  Not  only  is  this  matrix  constant  and  therefore  the  sign  of  its  determinant 
independent  of  surface  orientation,  but  there  is  a  nice  intuitive  relationship  between  the 
rotational  change  in  orientation  of  incident  point  light  source  illumination  and  the  induced 
rotational  change  in  image  intensity  gradient  with  respect  to  the  ratio  images: 


DEno  = 

A  positive  rotational  component  difference  between  {pi,qi)  and  {p2, 92)  makes  the  reflectance 
gradient  matrix  have  positive  determinant,  inducing  a  positive  rotation  in  the  image  intensity 
gradient  in  the  ratio  image  at  elliptical  points,  a  negative  rotation  at  hyperbolic  points,  and 
no  rotation  at  parabolic  points.  In  general,  for  a  non-zero  rotational  component  between 
incident  point  source  orientations  (pi,gi)  and  (^2,92))  the  image  intensity  gradient  in  the 
ratio  images  will  always  have  a  non-zero  rotational  change  in  the  same  direction  as  that  for 
the  point  sources  at  elliptical  points,  a  non-zero  rotational  change  retrograde  to  that  for  the 
point  sources  at  hyperbolic  points,  and  no  rotational  change  at  parabolic  points.  The  exact 
values  for  {pi,qi)  and  {p2, 92)  need  never  be  known,  only  their  relative  rotational  orientation 
about  the  viewing  direction.  The  only  computation  required  are  the  orientations  of  image 
intensity  gradients  in  the  ratio  images,  which  are  albedo  invariant. 
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If  the  third  light  source  incident  orientation  is  not  parallel  to  viewing  then  the  sign 
of  the  determinant  of  the  reflectance  map  gradient  matrix  using  the  Ratio-Of-Lambertian 
reflectance  map  will  not  be  invariant  for  all  surface  orientations.  There  is  however  good 
invariance  of  the  sign  of  the  determinant  over  a  wide  range  of  surface  orientations  if  the 
third  light  source  is  approximately  parallel  to  viewing-  a  slight  deviation  from  the  viewing 
direction  only  effects  a  very  small  range  of  points  at  occluding  contours  with  no  effect  at  all 
on  other  points.  The  restriction  of  having  the  third  light  source  parallel  to  viewing  is  not  at 
all  bad  considering  that  the  sign  of  Gaussian  curvature  only  has  a  chance  to  be  determined 
for  surface  orientations  that  are  mutually  illuminated  by  all  three  light  sources.  It  is  therefore 
preferable  to  have  one  of  the  light  sources  parallel  to  viewing  since  it  maximally  illuminates 
all  possible  surface  orientations  that  are  visible. 
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